Multi-fidelity Neural Architecture Search with Knowledge Distillation

Neural architecture search (NAS) targets at finding the optimal of a neural network for problem or family problems. Evaluations architectures are very time-consuming. One possible ways to mitigate this issue is use low-fidelity evaluations, namely training on part dataset, fewer epochs, with channels, etc. In paper, we propose Bayesian multi-fidelity method search: MF-KD. The relies new approach evaluations by few epochs using knowledge distillation. Knowledge distillation adds loss function term forcing mimic some teacher network. We carry out experiments CIFAR-10, CIFAR-100, and ImageNet-16-120. show that such modified leads better selection than logistic loss. proposed outperforms several state-of-the-art baselines.